Using physical location to modify behavior of a distributed virtual network element

ABSTRACT

A system for network virtualization in which physical network resources in different physical contexts are configured to implement one or more distributed logical network elements, at least some of the physical network resources implementing the distributed logical network elements configured according the physical context of those network resources. The local configuration of a physical locale is a version of the logical configuration that is modified specifically for the physical locale. Such modification is based on locale identifiers that are assigned to the physical locales. Some systems use locale-specific information to modify next-hop preference. Some system use locally modified configurations to determine the placement of VMs.

CLAIM OF BENEFIT TO PRIOR APPLICATIONS

The present application claims the benefit of U.S. Provisional PatentApplication 62/057,963, filed Sep. 30, 2014. U.S. Provisional PatentApplications 62/057,963 is incorporated herein by reference.

BACKGROUND

The benefits of network virtualization are well known. VMware® NSX® is aproduct suite that allows for virtualizing the network for VMs. NSXprovides the network elements as fundamental building blocks of adistributed virtual network, elements such as Distributed LogicalSwitches (for providing L2 based packet forwarding) and DistributedLogical Routers (for providing L3 based packet forwarding).

The notion of a distributed logical network element (e.g., a distributedlogical Switch or Router) is powerful since it allows the operator aconstruct a virtual network while hiding the underlying physical networkconnectivity and its limitations. The only thing required from theunderlying physical infrastructure is that it is capable of forwardingEthernet encapsulated IP frames.

The distributed logical network element is instantiated on a number ofphysical nodes (e.g. hypervisors) that participate in the logicalnetwork. These nodes also provide uniform packet forwarding capabilitiesin software. The control plane is responsible in configuring theseuniform policies on the participating hypervisor nodes. These policiesand configurations are necessarily at the logical level. In other words,they are not concerned with the underlying physical topology. Thisapproach works well where the underlying physical network provides“uniform connectivity” to all participating hypervisors. By “uniformconnectivity”, all the hypervisors are connected to a network withsimilar properties like latency, throughput, etc.

SUMMARY

Some embodiments of the invention provide a system for networkvirtualization in which physical network resources in different physicalcontexts are configured to implement one or more distributed logicalnetwork elements, at least some of the physical network resourcesimplementing the distributed logical network elements being configuredaccording the physical context of those network resources. In someembodiments, some of the distributed logical network elements arelogical forwarding elements such as distributed logical switches and/ordistributed logical routers. In some embodiments, the distributedlogical network elements are implemented by virtualization software orhypervisors running on host machines that are situated in differentphysical locales (e.g., sites or data centers).

In some embodiment, a local configuration of a physical locale is aversion of the logical configuration that is modified specifically forthe physical locale. In some embodiments, such modification is based onlocale identifiers that are assigned to the physical locales. In someembodiments, the local configuration is locally provided by the physicallocale itself. In some embodiments, the local configuration is providedby a centralized network controller or manager, which delivers localconfigurations to each of the physical locales. In some embodiments,local configurations are embedded in the logical configuration of theentire network, and it is up to a physical locale to identify portionsof the logical configuration as being applicable to that physicallocale. The networking and computing resources of the locale (e.g.,virtualization software in host machines) in turn uses the assignedlocale identifier to identify applicable local configuration embedded inthe logical configuration.

Different embodiments use locale-specific information differently tomodify the behavior of distributed logical network elements. Someembodiments use locale-specific information to modify next-hoppreference. Some embodiments perform ECMP to select the next hop fromVMs, MPSEs, and/or MPREs based on locale-specific information. In someembodiments, locally modified configurations are used to determine theplacement of VMs. In some embodiments, a VM placement engine uses localespecific information to decide the placement of VMs, i.e., to select asuitable host machine in a suitable physical locale hosting the VM basedon the locale-specific information of all the physical locales.

The preceding Summary is intended to serve as a brief introduction tosome embodiments of the invention. It is not meant to be an introductionor overview of all inventive subject matter disclosed in this document.The Detailed Description that follows and the Drawings that are referredto in the Detailed Description will further describe the embodimentsdescribed in the Summary as well as other embodiments. Accordingly, tounderstand all the embodiments described by this document, a full reviewof the Summary, Detailed Description and the Drawings is needed.Moreover, the claimed subject matters are not to be limited by theillustrative details in the Summary, Detailed Description and theDrawings, but rather are to be defined by the appended claims, becausethe claimed subject matters can be embodied in other specific formswithout departing from the spirit of the subject matters.

BRIEF DESCRIPTION OF THE DRAWINGS

The novel features of the invention are set forth in the appendedclaims. However, for purpose of explanation, several embodiments of theinvention are set forth in the following figures.

FIG. 1 illustrates a network in which distributed logical networkelements are implemented over physical elements that are each configuredaccording the physical contexts of those physical elements.

FIG. 2 conceptually illustrates the available networking and computingresources in the physical locales of the network.

FIG. 3 illustrates the virtualization software running in host machinesin the different locales.

FIG. 4 illustrates using assigned locale identifier to identifyapplicable portions of the logical configuration.

FIG. 5 conceptually illustrates the modification of logicalconfiguration into locale-specific configuration at the physicallocales.

FIGS. 6a-b conceptually illustrate processes for using physical locationinformation to modify the behaviors of distributed logical networkelements in logical configurations.

FIG. 7 conceptually illustrates using locale specific modification whenperforming ECMP for deciding the next hop of a distributed logicalnetwork elements.

FIG. 8 illustrates a VM placement engine that uses locale-specificinformation to decide the placement of VMs in a network.

FIG. 9 illustrates an example host machine that is operatingvirtualization software.

FIG. 10 conceptually illustrates an electronic system with which someembodiments of the invention are implemented.

DETAILED DESCRIPTION

In the following description, numerous details are set forth for thepurpose of explanation. However, one of ordinary skill in the art willrealize that the invention may be practiced without the use of thesespecific details. In other instances, well-known structures and devicesare shown in block diagram form in order not to obscure the descriptionof the invention with unnecessary detail.

Network virtualization is powerful because it frees the operator of thenetwork from having to actively manage the underlying physical networkconnectivity and limitations. However, in many instances, being aware ofits own physical context allows a hypervisor to optimize its ownperformance and throughput based on its physical context, even whileparticipating in a logical network element for network virtualization.Such physical context includes the hardware of the host machine thatimplements hypervisor, and of the local networking environment withinwhich the host machine is situated. For a host machine that is situatedin a data center, the infrastructure of the site also constitutes thephysical context of the hypervisor.

Some embodiments of the invention provide a system for networkvirtualization in which physical network resources in different physicalcontexts are configured to implement one or more distributed logicalnetwork elements, at least some of the physical network resourcesimplementing the distributed logical network elements being configuredaccording the physical context of those network resources. In someembodiments, some of the distributed logical network elements arelogical forwarding elements such as distributed logical switches and/ordistributed logical routers. In some embodiments, the distributedlogical network elements are implemented by virtualization software orhypervisors running on host machines that are situated in differentphysical locales (e.g., sites or data centers). FIG. 1 illustrates anetwork 100 in which distributed logical network elements areimplemented over physical elements that are each configured accordingthe physical contexts of those physical elements.

FIG. 1 includes a logical view 101 and a physical view 102 of thenetwork 100. The logical view of the network 100 is a logicalabstraction of the system that hides or encapsulates underlying physicalrealities of the network. Managing the network 100 through its logicalabstraction frees the operator of the network from having to manage theactual physical peculiarities of the network. In some embodiments, thelogical view 101 reflects a view of the network 100 as defined accordingto a logical configuration 105. The underlying physical infrastructureis in turn configured according to the logical configuration in order toimplement the network 100.

As illustrated, the logical view 101 (or the logical configuration) ofthe network includes virtual machines (VM) 111-118, a logical switch (L2forwarding elements) 121 for a network segment A, a logical switch 122for network segment B, a logical router/bridge (L3 forwarding element)131, and an edge gateway 141. The VMs 111-114 belong to the networksegment A and are interconnected by the logical switch 121. The VMs115-118 belong to the network segment B and are interconnected by thelogical switch 122. The network segments A and B (121 and 122) areinterconnected by the logical router/bridge 131, which is in turnconnected to the Internet through the edge gateway 141. In someembodiments, a network segment can be an IP subnet (such as VLAN) or anencapsulation overlay network such as VXLAN. In some embodiments, eachL2 network segment is identified by or referred to by a network segmentidentifier such as VNI (VXLAN network identifier). The logicalrouter/bridge 131 forwards packet between different network segments 121and 122. In some embodiments, the logical router/bridge 131 forwardpackets by L3 routing (e.g., using destination L3 IP address to look updestination VNI and destination L2 MAC address) and/or by bridging(using destination L2 MAC address to lookup destination VNI).

In some embodiments, the logical configuration is specified irrespectiveof the underlying physical realities. In other words, the logicalconfiguration is produced as if the connectivity of the underlyingphysical infrastructure is uniform and there is no distinction betweenresources in one locale versus resources in another locale. However, insome embodiments, the logical configuration is implemented over networkresources that are situated in different physical locations such thateach distributed logical network element (such as 121, 122, and 131) canspan across multiple physical locations.

The physical view 102 illustrates the underlying physical infrastructurethat implements the logical configuration of the network 100. Asillustrated, the network 100 is implemented over four physical locales191-194 (“SFO”, “LAX”, “SJC”, and “NYC”). The four physical locales191-194 are interconnected by inter-locale connections 180, which isprovided by the Internet in some embodiments. Each of these localesprovides a set of computing resources for implementing virtual machineas sell as networking resources for implementing logical switches androuters/bridges.

The logical switches 121 and 122, as well as the logical router/bridge131 of the logical configuration 105 span the four physical locales.Each of these logical forwarding elements (121, 122, and 131) areimplemented in a distributed fashion by the physical forwarding elementsin those four locales 191-194. Specifically, the logical switch 121 isimplemented by physical forwarding elements A1-A2 of locale 191, A3-A4of locale 192, A5-A6 of locale 193, and A7-A8 of locale 194. The logicalswitch 122 is implemented by physical forwarding elements B1-B2 oflocale 191, B3-B4 of locale 192, B5-B6 of locale 193, and B7-B8 oflocale 194. The logical router/bridge 131 is implemented by physicalforwarding elements R1-R2 of locale 191, R3-R4 of locale 192, R5-R6 oflocale 193, and R7-R8 of locale 194. In other words, the forwardingelements are implemented on networking and/or computing resources thatspan the four physical locales 191-194.

The VMs 111-118 (V1-V8) are likewise implemented on computing resourcesthat span the four physical locales. Specifically, the VMs 111-112 areimplemented by computing resources in “SFO”, the VMs 113-114 areimplemented by computing resources in “LAX”, the VMs 115-116 areimplemented by computing resources in the locale “SJC” (193), and theVMs 117-118 are implemented by computing resources in the locale “NYC”(194).

In some embodiments, each physical locale is a data center that includesits own collection of interconnected network resources. Each physicallocation has its own set of properties, capabilities and limitations.Such local properties can include the number and types of networking andcomputing resources available, the bandwidth of interconnectionsconnecting the networking resources, the latencies for sending datapackets between the networking and computing resources, etc. Thedifferent physical locales can also be geographically separated as toimpose significant latency for data to travel from one physical localeto another. The communication mediums that links the different physicallocations are likely of limited speed and throughput (e.g., theInternet) when compared with what is available within a data center orwithin a computing device.

Each physical locale is associated with a set of local configurationsthat is based on the properties, capabilities, and limitations of thephysical locale. When applied, the local configuration of a physicallocale modifies the behavior of networking and computing resources ofthe physical locale away from the uniformity specified by the logicalconfiguration in some embodiments. In some embodiments, the localconfiguration of a physical locale is based on locale specificproperties of the physical locale. This is unlike a logicalconfiguration in some embodiments, which hides the local peculiaritiesof the different locales from user and treats the entire network as auniform pool of resources and connections. In some embodiments, thelocal configuration of a locale distinguishes physical components in thelocale from physical components from other locales. In some embodiments,the local configuration prefers or forbids using certain networkresources under certain conditions. In some embodiments, the localconfiguration reserves network resources for traffic from certain nodes.In some embodiments, the local configuration is based on performancemetrics (such as latency, available memory, available bandwidth, etc.)of physical components in the network. In some embodiments, the localconfiguration is made to optimize resource usage in the physical localein order to balance load, avoid congested routing resources, guaranteebandwidth, or perform equal cost multi-path routing (ECMP).

As illustrated, the local configurations of the locales 191-194 (“SFO”,“LAX”, “SJC”, and “NYC”) are conceptually illustrated as having localconfigurations 151, 152, 153, and 154, respectively. These localconfigurations are unlike the logical configuration (i.e., the logicalview 101) of the network, which is applicable uniformly to all fourphysical locales. The local configuration 151 is applicable only toresources within the locale “SFO”. The local configuration 152 isapplicable only to resources within the locale “LAX”. The localconfiguration 153 is applicable only to resources within the locale“SJC”. The local configuration 154 is applicable only to resourceswithin the locale “NYC”.

FIG. 2 conceptually illustrates the available networking and computingresources in the physical locales 191-194 (“SFO”, “LAX”, “SJC”, “NYC”)of the network 100. As illustrated, each physical locale includes a setof computing devices labeled as “host” (hosts 211-213 in the locale 191,hosts 221-223 in the locale 192, hosts 231-233 in the locale 193, andhosts 241-243 in the locale 194). These computing devices are runningvirtualization software (labeled “VSW”) that allows the computingresource to serve as host machines of VMs. The virtualization softwarerunning in the host machines are configured according to the logicalconfiguration 105 of the network 100, which allows the host machines toinstantiate local copies of the logical switch 121-122 and of thelogical router/bridge 131 as physical forwarding elements. Some of thephysical locales also include networking devices labeled as “ToR” (Topof Rack) or “PH” (physical routers). These network devices performpacket forwarding tasks (L3/L2/bridging), but do not operatevirtualization software. The host machines, ToRs, and PHs of a physicallocale are interconnected by intra-locale connectivity of the physicallocale. The computing and routing resources of the physical locale arethen connected to the Internet through a gateway device labeled as“GWY”.

As mentioned, the computing and network resources of each physicallocale are configured according to a set of local configuration inaddition to the logical configuration of the network 100. Asillustrated, the computing and network resources (host machines, PH's,ToRs, etc) of the physical locales 191-194 are configured by localconfigurations 151-154, respectively. In some embodiments, the localconfigurations are used to configure the computing and networkingresource of each physical locale according to the local properties ofeach locale. For example, different locales can have different number ofcomputing devices that serve as host machines. Different locale can havedifferent types of computing devices that serve as host machines.Different locale can have different number or types of ToRs or PHs. Theintra-locale connectivity of one locale can have different capacity orlatency than the intra-locale connectivity of another locale. Andsending packets to a computing or networking resource of a same localethrough intra-locale connectivity generally requires less latency thanthrough the Internet to a computing or networking resource of adifferent locale.

FIG. 3 illustrates the virtualization software running in host machinesin the different locales. The virtualization software manages theoperations of the VMs as well as their access to the computing resourcesand the network resources of the host machines. In some embodiments, thevirtualization software provides an interface between each VM and alogical switch supported by the underlying network. Virtualizationsoftware may include one or more software components and/or layers,possibly including one or more of the software components known in thefield of virtual machine technology as virtual machine monitors (VMMs),hypervisors, or virtualization kernels. Because virtualizationterminology has evolved over time and has not yet become fullystandardized, these terms do not always provide clear distinctionsbetween the software layers and components to which they refer. As usedherein, the term, “hypervisor” is intended to generically refer to asoftware layer or component logically interposed between a virtualmachine and the host platform. Computing devices that serve as hostmachines will be further described in Section III below.

FIG. 3 illustrates four host machines 211, 221, 231, and 241 that are inlocales 191-194 (“SFO”, “LAX”, “SJC”, and “NYC”), respectively. Each ofthese host machines is running virtualization software that allows it tobe part of the logical configuration 105 of the network 100. Thevirtualization software allows its host machine to operate localinstantiations of the logical switches and logical routers/bridges inthe logical configuration. As mentioned, the logical configuration 105includes the logical switches 121-122 and the logical router/bridge 131.A local instantiation of the logical switch 121 in a host machine is aL2 physical switching element for the network segment A. A localinstantiation of the logical switch 122 in a host machine is a L2physical switching element for the network segment B. A localinstantiation of the logical routing/bridging element in a host machineis a physical routing or bridging element. As these physical forwardingelements are managed by the virtualization software, they are alsoreferred to as managed physical switching elements (MPSE) and managedphysical routing elements (MPRE). Further descriptions of MPSEs, MPREs,logical routing elements (LREs), and logical switching elements (LSEs)can be found in U.S. patent application Ser. No. 14/137,862 filed onDec. 20, 2013, titled “Logical Router”. U.S. patent application Ser. No.14/137,862 is herein incorporated by reference.

According to the logical configuration 105 of the network 100, the hostmachine 211 instantiates MPSEs 311 and 321 as physical forwardingelements A1, B1. The host machine 211 also instantiates an MPRE 331 asthe physical forwarding element R1. The physical forwarding elements A3,B3, R3, A5, B5, R5, A7, B7, R7 are likewise implemented byinstantiations of MPSEs and MPRE in the host machines 221, 231, and 241.(In some embodiments, MPSEs of different network segments in a same hostmachine are implemented as one MPSE in that host machine. Such an MPSEdistinguishes L2 traffic of different network segment by VNI identifiersin some embodiments).

In addition to the MPSEs and the MPREs, the virtualization software of ahost machine also implements local configurations that are specific tothe physical locale or to the host machine. In some embodiments, thevirtualization software of a host machine of a locale uses informationspecific to that locale to modify next hop decisions. As illustrated,the host machine 211 has information 341 specific to the locale “SFO”,and it uses this locale information to create or modify a set of nexthop tables 351. The MPSE 311 and the MPRE 321 in the host machines 211in turn use the modified next hop tables to select the destination forthe packets it produces. Likewise, the MPSEs and the MPREs in the hostmachines 221, 231, and 241 also uses their respective localconfiguration or locale information to determine next hop. In otherwords, the physical locale information (or local configuration) modifiesthe behaviors of the distributed logical network elements (MPSEs andMPREs) that are specified by the logical configuration.

Several more detailed embodiments of the invention are described below.Section I describes methods for implementing locale-specificconfiguration for modifying the behavior of distributed logical networkelements. Section II describes examples of using locale-specificconfigurations to modify behaviors of distributed logical networkelements. Section III describes an example communications device thatimplements some embodiments of the invention. Finally, section IVdescribes an electronic system with which some embodiments of theinvention are implemented.

I. Implementing Locale-Specific Configurations

Different embodiments implement local configurations specific tophysical locales (or locale-specific configurations) differently. Insome embodiment, a local configuration of a physical locale is a versionof the logical configuration modified specifically for the physicallocale. In some embodiments, such modification is based on localeidentifiers that are assigned to the physical locales. In someembodiments, the local configuration is locally provided by the physicallocale itself. In some embodiments, the local configuration is providedby a centralized network controller or manager, which delivers localconfigurations to each of the physical locales. In some embodiments,local configurations are embedded in the logical configuration of theentire network, and it is up to a physical locale to identify portionsof the logical configuration as being applicable to that physicallocale. The networking and computing resources of the locale (e.g.,virtualization software in host machines) in turn uses the assignedlocale identifier to identify applicable local configuration embedded inthe logical configuration.

An example portion of a logical configuration is a definition of a routeentry for a logical router (e.g., the logical router 131), which in someembodiments is a directive/command/definition/statement that looks like:

<destination-network/prefix> via nexthop <nh> forward on interface<id>  (1)

Thus, a logical configuration statement “0/0 via nexthop 192.168.1.1forward on interface lif0” would be globally applicable to all physicallocales. However, this definition does not account for any modifierbased on physical locales. A route entry having a locale based modifierlooks like:

<destination-network/prefix> via nexthop <nh> on locale <locale-id>forward on interface <id>  (2)

Such a route entry is a directive that allows locale-specific propertyto be taken into consideration. Thus, a route entry “0/0 via nexthop192.168.1.1 on locale ‘NYC’ forward on interface lif0” is a route entryhaving locale modifier “NYC”. It is a local configuration that would berecognized as a local configuration specific to locale “NYC” while beingignored by other locales based on the locale ID “NYC”. Likewise a routeentry “0/0 via nexthop 192.168.1.2 on locale ‘LAX’ forward on interfacelif0” is a route entry having local modifier “LAX”. It would berecognized as a local configuration specific to locale “LAX” while beingignored by other locales based on the locale ID “LAX”.

FIG. 4 illustrates using assigned locale identifier to identifyapplicable portions of the logical configuration. The figure illustratesthe logical configuration 105 for the network 100 that includes portionsapplicable to all physical locales as well as portions that are localespecific. As illustrated, the logical configuration 105 includes aglobally applicable portion 410 that is applicable to all locales. Thelogical configuration 105 also includes portions 411-414 that arespecific to locales 191-194 (“SFO”, “LAX”, “SJC”, and “NYC”),respectively. In some embodiments, such locale-specific portions of thelogical configuration include directives having locale specificmodifiers such as route entry (2) described above.

In order for each physical location to identify portions of the logicalconfiguration 105 that are applicable to them, a network manager 150first assigns locale identifiers to the physical locales. In someembodiment, each physical locale receives its locale identifier duringan initial configuration phase of the network 100. The physical localethen uses the receive locale ID to identify and apply applicableportions of the logical configuration while ignoring portions of thelogical configuration that is applicable only to other physical locales.In other words, each physical locale uses the physical locale identifieras a filter to accept and apply only relevant portions of the logicalconfiguration. For example, the locale “SFO” (191) uses its assignedlocale identifier to accept only the global configuration 410 and theportion 411 that is specific to the locale “SFO”. In some embodiments,portions of the logical configuration that are specific to a physicallocale are marked by the identifier of that physical locale, so eachphysical locale identifies the relevant portion of the logicalconfiguration by comparing its own received physical locale ID with thelocale IDs that marks the different portions of the logicalconfiguration. A process for identifying locale-specific portions oflogical configuration as local configuration is described by referenceto FIG. 6a below.

FIG. 4 also conceptually illustrates a filter for each of the physicallocales (locale filters 421-424 for locales 191-194, respectively),which uses the locale ID of its physical locale to filter-in relevantportions of the logical configuration for its locale. In someembodiments, the filtering of the logical configuration for a physicallocale is performed by the networking and computing resources within thelocale (such as by virtualization software of the host machines). Insome embodiments, the filtering of the logical configuration isperformed by a local configuration manager in each of the locales.

As mentioned, in some embodiment, a local configuration of a physicallocale is a version of the logical configuration that is modifiedspecifically for the physical locale. In some of these embodiments, atleast some of the modification is performed at the physical localesthemselves based on their locale-specific properties. FIG. 5conceptually illustrates the modification of logical configuration intolocale-specific configuration at the physical locales. As illustrated,the network manager 150 delivers the logical configuration 105 to eachof the physical locales 191-194. Each of the physical locales thenperforms local modification on the received logical configuration basedon information or properties specific to that locale to produce a set oflocal configuration specific to that locale. For example, the locale“SFO” (191) modifies logical configuration 105 into a SFO specific localconfiguration 511 based on SFO specific information 521. Themodification is performed by a local modification module 501, which insome embodiments is part of the virtualization software operating inhost machines of the physical locale 191.

The locale specific information (e.g., the “SFO” local information 521)in some embodiments includes portions of the logical configuration thatis relevant to the locale provided by the network manager 150, such asthe locale specific portions 411-414 of the logical configuration or thelocale specific route entry for a logical router as described above. Thelocale specific information in some embodiments includes static networktopology of the physical locale (e.g., a data center), which is providedby a central network manager in some embodiments in which the networkmanager has such information. In some embodiments, the locale specificinformation is furnished locally by a local network manager (e.g., thelocal manager 531 of the physical locale “SFO” 191). In someembodiments, the locally furnished information includes static networktopology of the physical locale. In some embodiments, the locallyfurnished information includes dynamic network metrics of the physicallocale that are updated in real time (such as latency, available memory,available bandwidth, etc.) A process for modifying logical configurationbased on locale-specific information is described by reference to FIG.6b below.

For some embodiments, FIGS. 6a-b conceptually illustrate processes forusing physical location information to modify the behaviors ofdistributed logical network elements in logical configurations. FIG. 6aconceptually illustrates a process 601 for using physical localeidentifier to identify locale-specific portions of the logicalconfiguration (or configuration definitions having a specific localemodifier). FIG. 6b conceptually illustrates a process 602 for modifyinglogical configuration based on locale specific information. In someembodiments, the processes 601 and 602 are performed by thevirtualization software of a host machine in a physical locale.

Some embodiments perform the process 601 during initial configuration ofa network according to a logical configuration. For example, the hostmachines (i.e., their virtualization software) in the physical locales191-194 perform the process 601 when the network manager 150 isconfiguring the underlying physical hardware according to the logicalconfiguration 105.

The process 601 starts when it receives (at 610) a locale identifierfrom the network manager. The locale identifier is the identifier thatthe network manager assigns to the physical locale that includes hostmachine running the process. The process then receives (at 620) logicalconfiguration from the network manager. In some embodiments, the processreceives the locale identifier and the logical configuration throughcontrol plane communications with the network manager.

Next, the process identifies (at 630) applicable portions of the logicalconfiguration based on the received locale ID that is assigned to thephysical locale of the host machine. In some embodiments, the processfilters the logical configuration to accept only relevant portions ofthe logical configuration, i.e., the portions that are either globallyapplicable to all physical locations or the portions that are marked bythe matching locale ID. In some embodiments, the relevant portions ofthe logical configuration are configurationdefinitions/statements/directives that are modified by a locale ID(e.g., the locale-ID-modified route entry (2)).

Finally, the process configures (at 640) the distributed logical networkelements (e.g., MPSE and MPRE) by applying the portions of the logicalconfiguration identified as relevant to this physical locale. In someembodiments, the globally applicable portions of the logicalconfiguration instantiates the distributed logical network elements,while the locale-specific portions of the logical configuration modifiesthe behavior of the instantiated distributed logical network elements.The process 601 then ends.

The process 601 is a process for configuring a physical locale based oninformation provided by a network manager during an initialconfiguration of the network. The process 602 is on the other hand aprocess for modifying a received logical configuration. Some embodimentsperforms the process 602 after the initial configuration by modifyingthe received logical configuration based on locale specific information,which in some embodiments includes network topology from the perspectiveof the physical locale and/or static/dynamic network performance metricsthat are provided by the physical locale. Some embodiments perform theprocess 602 during or soon after the initial configuration.

The process 602 receives (at 650) the logical configuration from networkmanager during initial configuration. The process 602 then identifies(at 660) the locale specific information. In some embodiments, theprocess 602 identifies the locale specific information within theconfiguration data transmitted by the central network manager. In someembodiments, the process 602 identifies the locale specific informationfrom information that are available locally at the physical locale, suchas dynamic real-time performance matric of the locale or the networktopology of the locale.

The process then modifies (at 670) the logical configuration based onthe identified locale specific information. The process then configures(at 680) the distributed logical network elements in the logicalconfiguration by applying the modified logical configuration. Forexample, in some embodiments, the process modifies the logicalconfiguration by modifying next hop preferences of a MPSE or MPRE basedon identified locale information (hence modifying the MPSE/MPRE'sbehavior). The locale-specific information in some embodiments includesinformation on networking resources in the locale that allows adistributed logical network element to select or avoid a particular nodeas next hop. After applying the modified logical configuration, theprocess 602 ends.

II. Use Cases of Locale-Specific Configurations

Different embodiments use locale-specific information differently tomodify the behavior of distributed logical network elements. Forexample, as mentioned, some embodiments use locale-specific informationto modify next-hop preferences. Specifically, some embodiments performECMP to select the next hop from VMs, MPSE, and/or MPREs. Thelocale-specific information or modified logical configuration allows thenext-hop selection for these globally configured distributed logicalnetwork elements to operate based on local information that are hiddenfrom the central operator of the network.

For some embodiments, FIG. 7 conceptually illustrates using localespecific modification when performing ECMP for deciding the next hop ofa distributed logical network elements. The figure illustrates themodification of the logical configuration for the VM 112 (V2) and theMPSE 311 (A1) based on locale-specific information of the locale 191“SFO” (since both V2 and A1 are instantiated in host machines of thelocale 191).

As illustrated, the logical configuration 105 of the network 100includes, among others, next hop table 701 for the VM 112 (V2) and nexthop table 702 for the MPSE 311 (A1). The VM 112 belongs to networksegment B, and hence all MPSEs of the logical switch B 122 (B1-B8) areeligible as next hop for the VM 112. The MPSE 311 (A1) is aninstantiation of the logical switch element 121 for the network segmentA (logical switch A). Since the logical router 131 is a next hop of thelogical switch A, all of the MPREs belonging to the logical router 131(R1-R8) are eligible as next hop for MPSE 311. However, these eligiblehops are determined strictly from the logical configuration point ofview without considering any locale-specific information.

The content of the logical configuration 111 is processed by the localespecific modification module 501 of the locale “SFO”. The module 501uses information specific to the locale “SFO” (local information 521) tomodify the logical configuration 105 for the distributed logical networkelements (MPSEs and MPREs) running in the host machines of the physicallocale “SFO”. In some embodiments (not illustrated), the locale specificmodification is based on relevant portions of the logical configurationthat are identified by a locale ID assigned by the network manager. Thelocale specific modification 501 produces a locale-modifiedconfiguration 711 for the VM 112 (V2) and a locale-modifiedconfiguration 712 for the MPSE 311 (A1)

The locale-modified configurations distinguish between the possibledifferent next hops of a distributed logical network element.Specifically, the modified configuration 711 distinguishes betweenpossible next hops B1-B8 based on their separation from the VM 112 (V2),while the modified configuration 712 distinguishes between the possiblenext hops R1-R8 based on their separation from the MPSE311 (A1). Suchdistinctions are made possible by the locale specific information 521,which identifies next hops in this physical locale/data center versusnext hops in other physical locales/data center. For example, themodified configuration 711 identifies B1 and B2 as next hops in the samedata center as V2 (also in locale “SFO”), B5 and B6 as next hops in anear data center (in locale “SJC”), B3, B4, B7, and B8 as next hops thatare either far or very far (local “LAX” or “NYC”). Based on theinformation, a ECMP module 721 for the VM 112 will be able to next hopselection decisions that are optimized based on the locale specificinformation (e.g., by favoring or weighing toward next hops that arenear or in the same data center). The ECMP module 722 likewise makesnext hop decisions based on locale information that distinguishesbetween R1-R8 as next hops (R1 and R2 being in the same data center asA1, R5 and R6 being in a near data center, R3, R4, R7, R8 being in fardata center).

The modified configuration 712 is an example local configuration thatuses locale-specific parameters (i.e., geographic separation from thecurrent locale) to characterize the possible next hops. FIG. 7 alsoillustrates other example modified configurations (762 and 772) that useother locale-specific parameters to characterize the possible next hops.

The modified configuration 762 reports the latencies to/from thepossible next hops R1-R8 of the MPSE311 (A1). The ECMP module 722 canthen use the reported latencies to select the next hop. As illustrated,next hops R1 and R2 has the smallest latencies (possibly because theyare in the same locale “SFO”), while next hops R7 and R8 has the largestlatencies (possibly because they are in a distant locale “NYC”). In someembodiments, the latency metrics reported in the modified configuration762 are based on real-time measurements obtained by the physicalhardware of the locale. In some embodiments, the latency metricsreported are historical averages that can be supplied locally by datacenter or centrally by the central network manager (e.g., 150).

The modified configuration 772 specifies directives or commands that areassociated with each of the possible next hops. As illustrated, R3, R4,R7, and R8 are specified as “forbidden” for traffic from MPSE A1, whileR2 is specified as “preferred”. R1 has a directive that says “reservedfor V1” such that only traffic originated from the VM 111 (V1) may useR1 as next hop. Directives such as these are used by some embodiments toguarantee bandwidth for a particular VM by e.g., reserving a networkingresource for traffic from the particular VM. Such directives can bespecified locally by the data center or centrally by the network manager(e.g., 150) in some embodiments.

In some embodiments, modified configurations of different types are usedjointly by ECMP modules when selecting a next hop. For example, the ECMPmodule 722 can use the information from any or all of the modifiedconfigurations 712, 762, and 772 when selecting a next hop. Though notillustrated, other locale-specific parameters can also be used tocharacterize next hops (e.g., storage capacity of computing device,processor speed, etc.). More generally, different locale-specificparameters can be used to modify the behaviors of distributed logicalnetwork elements or to modify uniform/global logical configurations thatare centrally specified by a network manager.

In some embodiments, locally modified configurations are used todetermine the placement of VMs. As mentioned, a logical network may spanmultiple data centers in different geographical locations. In someembodiments, the operator or the user of the logical network does notspecify or care which host machine in which data center is hosting theVM, but a VM placement engine uses locale specific information to decidethe placement of VMs, (i.e., to select a suitable host machine in asuitable physical locale hosting the VM based on the locale-specificinformation of all the physical locales).

For some embodiments, FIG. 8 illustrates a VM placement engine 810 thatuses locale-specific information to decide the placement of VMs in anetwork. The figure illustrates the locale-specific information based VMplacement in two stages 801-802.

At the first stage 801, the VM placement engine 810 receiveslocale-specific information 521-524 for the locales 191-194 (“SFO”,“LAX, “SJC”, and “NYC”). In some embodiments, such information aregenerated and provided by the physical locales (data centers)themselves. In some embodiments such information are maintained andprovided centrally by the network manager (e.g., 150). The VM placementengine 810 also receives a list of VMs that needs to be placed from thelogical configuration 105 of the network 100.

At the second stage 802, the VM placement engine 810 assigns the VMsspecified by the logical configuration to host machines in the differentphysical locales. The VM placement engine examines the locale specificinformation when selecting a locale and a host machine for a VM. In someembodiments, some or all of the VMs in the logical configuration areeach associated with a unique set of requirements, and the placementengine in turn finds the most suitable host machine and physical localefor those VMs based on the VM's requirements and the locale's specificproperties. For example, a particular VM may require certain bandwidthguarantee, and the VM placement engine would in turn use thelocale-specific information to identify a physical locale and a hostmachine that can provide the requisite bandwidth. Such locale-specificinformation may in some embodiments include an identification of aparticular distributed logical network element (MPSE or MPRE) running ona particular host machine that has sufficient bandwidth to handle theguaranteed bandwidth requirement.

Though not illustrated, in some embodiments, the VM placement enginesends directives to each of the locales after the placements of the VMsare decided. In some embodiments, such directives are alsolocale-specific, such as to reserve a particular networking or computingresource in the locale for a particular VM, or to require a particularnext hop for a particular distributed logical network element, etc. Insome embodiments, the delivery of such directives or command relies onlocale IDs that were assigned to the locales by the network manager asdescribed by reference to FIG. 4 above.

III. Computing Device

As mentioned earlier, some embodiments of the invention are implementedby virtualization software or hypervisors running on computing devicesserving as host machines. For some embodiments, FIG. 9 illustrates anexample host machine 900 that is operating virtualization software 905.The virtualization software 905 allows the host machine to host virtualmachines 911-914 as well as connecting the virtual machines to aphysical network 990. This physical network 990 may span one or moredata centers and include various physical switches and routers.

As illustrated, the host machine 900 has access to the physical network990 through a physical NIC (PNIC) 995. The virtualization software 905serves as the interface between the hosted VMs 911-914 and the physicalNIC 995 (as well as other physical resources, such as processors andmemory). Each of the VMs includes a virtual NIC (VNIC) for accessing thenetwork through the virtualization software 905. Each VNIC in a VM isresponsible for exchanging packets between the VM and the virtualizationsoftware 905. In some embodiments, the VNICs are software abstractionsof physical NICs implemented by virtual NIC emulators.

The virtualization software 905 manages the operations of the VMs911-914, and includes several components for managing the access of theVMs to the physical network (by implementing the logical networks towhich the VMs connect, in some embodiments). As illustrated, thevirtualization software 905 includes a physical switching element 920, aphysical routing element 930, a controller interface 940, an uplinkmodule 970, and a locale-specific configuration module 950.

The controller interface 940 receives control plane messages from acontroller or a cluster of controllers 960. In some embodiments, thesecontrol plane message includes configuration data for configuring thevarious components of the virtualization software and/or the virtualmachines (such as the physical switching element 920 and the physicalrouting element 930). In some embodiments, the control plane messagesalso include locale-specific configuration information from a centralnetwork manager or a local network manager.

The physical switching element 920 (or managed physical switchingelement, MPSE) delivers network data to and from the physical NIC 995,which interfaces the physical network 990. The physical switchingelement also includes a number of virtual ports (vPorts) thatcommunicatively interconnects the physical NIC with the VMs 911-914, thephysical routing element 930 and the controller interface 940. Eachvirtual port is associated with a unique L2 MAC address, in someembodiments. The physical switching element performs L2 link layerpacket forwarding between any two network elements that are connected toits virtual ports. The physical switching element also performs L2 linklayer packet forwarding between any network element connected to any oneof its virtual ports and a reachable L2 network element on the physicalnetwork 990 (e.g., another VM running on another host).

The physical routing element 930 (or managed physical routing element,MPRE) performs L3 routing (e.g., by performing L3 IP address to L2 MACaddress resolution) on data packets received from a virtual port on thephysical switching element 920. In some embodiments, the virtual portthat the physical routing element 930 is attached to is a sink port.Each routed data packet is then sent back to the physical switchingelement 920 to be forwarded to its destination according to the resolvedL2 MAC address. This destination can be another VM connected to avirtual port on the physical switching element 920, or a reachable L2network element on the physical network 990 (e.g., another VM running onanother host, a physical non-virtualized machine, etc.).

The locale specific configuration module 950 stores information specificto the physical locale of the host machine. The information is madeavailable to the physical switching element 920 and the physical routingand bridging element 930, which uses the locale specific information tomodify their behavior by e.g., favoring or disfavoring specific nexthops. In some embodiments, the locale specific configuration module usesa locale identifier to determine if data received from controllerinterface 940 is a locale-specific configuration data that is applicableto this host machine. The locale specific configuration module 950 insome of these embodiments act as a filter and allow only relevantconfiguration data to be used to modify the behavior of the physicalswitching element 920 and the physical routing element 930.

The uplink module 970 relays data between the physical switching element920 and the physical NIC 995. In some embodiments, the uplink module 970allows the host machine 900 to serve as a tunnel endpoint forencapsulation overlay networks such as VXLAN and VLANs. VXLAN is anoverlay network encapsulation protocol. An overlay network created byVXLAN encapsulation is sometimes referred to as a VXLAN network, orsimply VXLAN. When a VM on the host 900 sends a data packet (e.g., anethernet frame) to another VM in the same VXLAN network but on adifferent host, the uplink module 970 encapsulates the data packet usingthe VXLAN network's VNI and network addresses of the VTEP, beforesending the packet to the physical network. The packet is tunneledthrough the physical network (i.e., the encapsulation renders theunderlying packet transparent to the intervening network elements) tothe destination host. The uplink module 970 also decapsulates incomingVXLAN packets and forwards only the original inner data packet to thedestination VM.

In this document, the term “packet” refers to a collection of bits in aparticular format sent across a network. One of ordinary skill in theart will recognize that the term packet may be used herein to refer tovarious formatted collections of bits that may be sent across a network,such as Ethernet frames, TCP segments, UDP datagrams, IP packets, etc.

This specification refers throughout to computational and networkenvironments that include virtual machines (VMs). However, virtualmachines are merely one example of data compute nodes (DCNs) or datacompute end nodes, also referred to as addressable nodes. DCNs mayinclude non-virtualized physical hosts, virtual machines, containersthat run on top of a host operating system without the need for ahypervisor or separate operating system, and hypervisor kernel networkinterface modules.

VMs, in some embodiments, operate with their own guest operating systemson a host using resources of the host virtualized by virtualizationsoftware (e.g., a hypervisor, virtual machine monitor, etc.). The tenant(i.e., the owner of the VM) can choose which applications to operate ontop of the guest operating system. Some containers, on the other hand,are constructs that run on top of a host operating system without theneed for a hypervisor or separate guest operating system. In someembodiments, the host operating system uses name spaces to isolate thecontainers from each other and therefore provides operating-system levelsegregation of the different groups of applications that operate withindifferent containers. This segregation is akin to the VM segregationthat is offered in hypervisor-virtualized environments that virtualizesystem hardware, and thus can be viewed as a form of virtualization thatisolates different groups of applications that operate in differentcontainers. Such containers are more lightweight than VMs.

Hypervisor kernel network interface modules, in some embodiments, is anon-VM DCN that includes a network stack with a hypervisor kernelnetwork interface and receive/transmit threads. One example of ahypervisor kernel network interface module is the vmknic module that ispart of the ESXi™ hypervisor of VMware, Inc.

One of ordinary skill in the art will recognize that while thespecification refers to VMs, the examples given could be any type ofDCNs, including physical hosts, VMs, non-VM containers, and hypervisorkernel network interface modules. In fact, the example networks couldinclude combinations of different types of DCNs in some embodiments.

IV. Electronic System

Many of the above-described features and applications are implemented assoftware processes that are specified as a set of instructions recordedon a computer readable storage medium (also referred to as computerreadable medium). When these instructions are executed by one or moreprocessing unit(s) (e.g., one or more processors, cores of processors,or other processing units), they cause the processing unit(s) to performthe actions indicated in the instructions. Examples of computer readablemedia include, but are not limited to, CD-ROMs, flash drives, RAM chips,hard drives, EPROMs, etc. The computer readable media does not includecarrier waves and electronic signals passing wirelessly or over wiredconnections.

In this specification, the term “software” is meant to include firmwareresiding in read-only memory or applications stored in magnetic storage,which can be read into memory for processing by a processor. Also, insome embodiments, multiple software inventions can be implemented assub-parts of a larger program while remaining distinct softwareinventions. In some embodiments, multiple software inventions can alsobe implemented as separate programs. Finally, any combination ofseparate programs that together implement a software invention describedhere is within the scope of the invention. In some embodiments, thesoftware programs, when installed to operate on one or more electronicsystems, define one or more specific machine implementations thatexecute and perform the operations of the software programs.

FIG. 10 conceptually illustrates an electronic system 1000 with whichsome embodiments of the invention are implemented. The electronic system1000 can be used to execute any of the control, virtualization, oroperating system applications described above. The electronic system1000 may be a computer (e.g., a desktop computer, personal computer,tablet computer, server computer, mainframe, a blade computer etc.),phone, PDA, or any other sort of electronic device. Such an electronicsystem includes various types of computer readable media and interfacesfor various other types of computer readable media. Electronic system1000 includes a bus 1005, processing unit(s) 1010, a system memory 1025,a read-only memory 1030, a permanent storage device 1035, input devices1040, and output devices 1045.

The bus 1005 collectively represents all system, peripheral, and chipsetbuses that communicatively connect the numerous internal devices of theelectronic system 1000. For instance, the bus 1005 communicativelyconnects the processing unit(s) 1010 with the read-only memory 1030, thesystem memory 1025, and the permanent storage device 1035.

From these various memory units, the processing unit(s) 1010 retrievesinstructions to execute and data to process in order to execute theprocesses of the invention. The processing unit(s) may be a singleprocessor or a multi-core processor in different embodiments.

The read-only-memory (ROM) 1030 stores static data and instructions thatare needed by the processing unit(s) 1010 and other modules of theelectronic system. The permanent storage device 1035, on the other hand,is a read-and-write memory device. This device is a non-volatile memoryunit that stores instructions and data even when the electronic system1000 is off. Some embodiments of the invention use a mass-storage device(such as a magnetic or optical disk and its corresponding disk drive) asthe permanent storage device 1035.

Other embodiments use a removable storage device (such as a floppy disk,flash drive, etc.) as the permanent storage device. Like the permanentstorage device 1035, the system memory 1025 is a read-and-write memorydevice. However, unlike storage device 1035, the system memory is avolatile read-and-write memory, such a random access memory. The systemmemory stores some of the instructions and data that the processor needsat runtime. In some embodiments, the invention's processes are stored inthe system memory 1025, the permanent storage device 1035, and/or theread-only memory 1030. From these various memory units, the processingunit(s) 1010 retrieves instructions to execute and data to process inorder to execute the processes of some embodiments.

The bus 1005 also connects to the input and output devices 1040 and1045. The input devices enable the user to communicate information andselect commands to the electronic system. The input devices 1040 includealphanumeric keyboards and pointing devices (also called “cursor controldevices”). The output devices 1045 display images generated by theelectronic system. The output devices include printers and displaydevices, such as cathode ray tubes (CRT) or liquid crystal displays(LCD). Some embodiments include devices such as a touchscreen thatfunction as both input and output devices.

Finally, as shown in FIG. 10, bus 1005 also couples electronic system1000 to a network 1065 through a network adapter (not shown). In thismanner, the computer can be a part of a network of computers (such as alocal area network (“LAN”), a wide area network (“WAN”), or an Intranet,or a network of networks, such as the Internet. Any or all components ofelectronic system 1000 may be used in conjunction with the invention.

Some embodiments include electronic components, such as microprocessors,storage and memory that store computer program instructions in amachine-readable or computer-readable medium (alternatively referred toas computer-readable storage media, machine-readable media, ormachine-readable storage media). Some examples of such computer-readablemedia include RAM, ROM, read-only compact discs (CD-ROM), recordablecompact discs (CD-R), rewritable compact discs (CD-RW), read-onlydigital versatile discs (e.g., DVD-ROM, dual-layer DVD-ROM), a varietyof recordable/rewritable DVDs (e.g., DVD-RAM, DVD-RW, DVD+RW, etc.),flash memory (e.g., SD cards, mini-SD cards, micro-SD cards, etc.),magnetic and/or solid state hard drives, read-only and recordableBlu-Ray® discs, ultra density optical discs, any other optical ormagnetic media, and floppy disks. The computer-readable media may storea computer program that is executable by at least one processing unitand includes sets of instructions for performing various operations.Examples of computer programs or computer code include machine code,such as is produced by a compiler, and files including higher-level codethat are executed by a computer, an electronic component, or amicroprocessor using an interpreter.

While the above discussion primarily refers to microprocessor ormulti-core processors that execute software, some embodiments areperformed by one or more integrated circuits, such as applicationspecific integrated circuits (ASICs) or field programmable gate arrays(FPGAs). In some embodiments, such integrated circuits executeinstructions that are stored on the circuit itself.

As used in this specification, the terms “computer”, “server”,“processor”, and “memory” all refer to electronic or other technologicaldevices. These terms exclude people or groups of people. For thepurposes of the specification, the terms display or displaying meansdisplaying on an electronic device. As used in this specification, theterms “computer readable medium,” “computer readable media,” and“machine readable medium” are entirely restricted to tangible, physicalobjects that store information in a form that is readable by a computer.These terms exclude any wireless signals, wired download signals, andany other ephemeral signals.

While the invention has been described with reference to numerousspecific details, one of ordinary skill in the art will recognize thatthe invention can be embodied in other specific forms without departingfrom the spirit of the invention. In addition, a number of the figures(including FIGS. 6a-b ) conceptually illustrate processes. The specificoperations of these processes may not be performed in the exact ordershown and described. The specific operations may not be performed in onecontinuous series of operations, and different specific operations maybe performed in different embodiments. Furthermore, the process could beimplemented using several sub-processes, or as part of a larger macroprocess. Thus, one of ordinary skill in the art would understand thatthe invention is not to be limited by the foregoing illustrativedetails, but rather is to be defined by the appended claims.

1-20. (canceled)
 21. A method of deploying machines in a plurality ofdatacenters, the method comprising: based on a set of placementcriteria, identifying a first datacenter for deploying a first machinesand deploying the first machine in the first datacenter; based on theset of placement criteria, identifying a second datacenter for deployinga second machines and deploying the second machine in the seconddatacenter; configuring a set of forwarding elements in the first andsecond datacenters to implement a logical network to connect the firstand second machines.
 22. The method of claim 21, wherein the logicalnetwork comprises at least two logical switches each of which spans thefirst and second datacenter, and a logical router that spans the firstand second datacenters.
 23. The method of claim 21, wherein theidentifying and deploying are performed by a placement engine, themethod further comprising receiving, at the placement engine,identification of the first and second machines as machines that have tobe deployed without specification of one of the datacenters as thedatacenter for deploying either first or second machine.
 24. The methodof claim 23 further comprising receiving, at the placement engine, firstand second locale information regarding the first and seconddatacenters, wherein identifying the first and second datacenters forthe first and second machines comprises using the first and secondlocale information to select the first and second datacentersrespectively for the first and second machines.
 25. The method of claim24, wherein the locale information comprises data regarding operationalmetrics of the forwarding elements in the first and second datacenters.26. The method of claim 24, wherein identifying the first and seconddatacenters for the first and second machines further comprises usingfirst and second sets of requirements for the first and second machinesto select the first and second datacenters respectively for the firstand second machines.
 27. The method of claim 21 further comprisingsending directives to first and second datacenters to reserve particularnetwork resources in the first and second datacenters for the first andsecond machines respectively.
 28. The method of claim 21 furthercomprising generating forwarding rules to configure forwarding elementsin the first and second datacenters to implement a logical forwardingelement in the logical network, and distributing the forwarding rules tothe forwarding elements in the first and second datacenters.
 29. Themethod of claim 28, wherein the forwarding rules specify preferred,forbidden, and reserved forwarding elements in the first and seconddatacenters to use for forwarding packets through the logical forwardingelement.
 30. The method of claim 29, wherein the forwarding rules arenext-hop forwarding rules to use for multi-path forwarding operations.31. A non-transitory machine readable medium storing a program forexecution by at least one processor, the program for deploying machinesin a plurality of datacenters, the program comprising sets ofinstructions for: based on a set of placement criteria, identifying afirst datacenter for deploying a first machines and deploying the firstmachine in the first datacenter; based on the set of placement criteria,identifying a second datacenter for deploying a second machines anddeploying the second machine in the second datacenter; directing a setof forwarding elements in the first and second datacenters to beconfigured to implement a logical network to connect the first andsecond machines.
 32. The non-transitory machine readable medium of claim31, wherein the logical network comprises at least two logical switcheseach of which spans the first and second datacenter, and a logicalrouter that spans the first and second datacenters.
 33. Thenon-transitory machine readable medium of claim 31, wherein the programis a placement engine, the program further comprising a set ofinstructions for receiving, at the placement engine, identification ofthe first and second machines as machines that have to be deployedwithout specification of one of the datacenters as the datacenter fordeploying either first or second machine.
 34. The non-transitory machinereadable medium of claim 33, the program further comprising a set ofinstructions for receiving, at the placement engine, first and secondlocale information regarding the first and second datacenters, whereinthe set of instructions for identifying the first and second datacentersfor the first and second machines comprises a set of instructions forusing the first and second locale information to select the first andsecond datacenters respectively for the first and second machines. 35.The non-transitory machine readable medium of claim 34, wherein thelocale information comprises data regarding operational metrics of theforwarding elements in the first and second datacenters.
 36. Thenon-transitory machine readable medium of claim 34, wherein the set ofinstructions for identifying the first and second datacenters for thefirst and second machines further comprises a set of instructions forusing first and second sets of requirements for the first and secondmachines to select the first and second datacenters respectively for thefirst and second machines.
 37. The non-transitory machine readablemedium of claim 31, the program further comprising a set of instructionsfor sending directives to first and second datacenters to reserveparticular network resources in the first and second datacenters for thefirst and second machines respectively.
 38. The non-transitory machinereadable medium of claim 31, the program further comprising sets ofinstructions for generating forwarding rules to configure forwardingelements in the first and second datacenters to implement a logicalforwarding element in the logical network, and distributing theforwarding rules to the forwarding elements in the first and seconddatacenters.
 39. The non-transitory machine readable medium of claim 38,wherein the forwarding rules specify preferred, forbidden, and reservedforwarding elements in the first and second datacenters to use forforwarding packets through the logical forwarding element.
 40. Thenon-transitory machine readable medium of claim 39, wherein theforwarding rules are next-hop forwarding rules to use for multi-pathforwarding operations.